Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

Daniel Nashed

Tags

First look - Ubuntu LTS 26.04 (Resolute Raccoon)

Daniel Nashed – 12 December 2025 21:10:25

Ubuntu is the next major distribution shipping their next long term release.
It's scheduled be shipped end of April. An early release is already available and I took a quick look.

The most simple test is to use the container image, which is already available.
But there is also a full ISO.

Here is the basic data.
The kernel is bumped up and also glibc version is 2.42 like what we have seen at Debian 13.
Because of Java dependencies discussed already for Debian 13 the first Domino Java version that will work is currently in Domino 14.5.1 shipping in the same time frame.
Hostname : nsh-ubuntu26 Linux OS : Ubuntu Resolute Raccoon (development branch) Linux Version : 26.04 Kernel : 6.17.0-6-generic GNU libc : 2.42 Timezone : UTC +0000 Locale : en_US.UTF-8

OpenSSL and curl also has been update to a much never version.
What I see today is:
OpenSSL 3.5.3 16 Sep 2025 (Library: OpenSSL 3.5.3 16 Sep 2025)
curl 8.17.0 (x86_64-pc-linux-gnu) libcurl/8.17.0 OpenSSL/3.5.3 zlib/1.3.1 brotli/1.1.0 zstd/1.5.7 libidn2/2.3.8 libpsl/0.21.2 libssh2/1.11.1 nghttp2/1.64.0 librtmp/2.3 mit-krb5/1.21.3 OpenLDAP/2.6.10 Release-Date: 2025-11-05, security patched: 8.17.0-1ubuntu1 Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

NGINX isn't the latest version. But on a good level.
nginx version: nginx/1.28.0 (Ubuntu)

The current version is 1.29.4.

This is a great update. And it aligns Ubuntu with the latest Debian 13 version.
Which is newer than the reference Redhat Enterprise 10.

There is one detail that needs to get fixed in the container image. "rev" used by the container scripts moved to a different packet.
A normal Ubuntu installation will not show a difference. But in a container we have to install the package. But this is already addressed in the container image build.

Here are the details. The mascot image isn't ready yet.

Download:
https://releases.ubuntu.com/26.04-snapshot1/

Draft Release Notes:
https://discourse.ubuntu.com/t/resolute-raccoon-release-notes/59221

Comments [0]

Regression in 14.5 - SPR # ASHEDNU9C2 XPages doesn’t preserve sessionScope variables across operations

Daniel Nashed – 12 December 2025 15:20:08

If you are using XPages applications this issue might hit you. I was about to update the DNUG production server to 14.5 FP1 and checked dependencies with the team responsible for applications.
One of the applications uses XPages and the business partner came back wit this TN and SPR

https://support.hcl-software.com/csm?id=kb_article&sysparm_article=KB0127489

I opened a ticket and got the info that a fix is currently being tested. I will provide an update once a public fix is available.
If you have already updated and run into this issue, you should open a ticket to get the test fix.

If you are running XPages I would probably wait until this fix is confirmed before updating application servers.

-- Daniel

Comments [0]

SFTP Linux

Mission impossible: sftp client with user/password support on Redhat UBI

Daniel Nashed – 12 December 2025 12:07:23

Redhat UBI isn't really my favorite container image.
It is licence friendly for container images. But that's it with the benefits.

Here is my current challenge. And this really shows that Linux is not always the same.
I am trying to build a SFTP upload with user/password.

There are two easy to use ways. Plus one other way with sshpass.
But none those methods work on UBI.

In short:

curl is compiled without sftp support
lftp is not available in UBI
scp alone does only support sftp with keys not user/password
sshpass is not available on UBI

I don't want to switch the container default image to a different based image, but UBI isn't really what helps me to provide a solution to use SFTP to upload diagnostic data to HCL support automatically.
Here are two ways that would work with Ubuntu and other base images. But this isn't available on UBI. curl -unotes:notes -T /etc/hosts sftp://127.0.0.1/home/notes/hosts lftp -u notes,notes sftp://127.0.0.1 -e "put /etc/hosts; bye"

I really don't want a solution that doesn't work for the standard base image used by HCL and the community project (UBI).
Probably the safest bet will be to write a small sftp client using libssh2 and statically link it on Alpine to make it available independent from the distribution used.
Similar to what I do with nshmailx.
This is less a technical challenge development a solution. It's more a bundling/tool chain issue.

Here is a quick summary created by ChatGTP to show the the challenge.
Probably I could write something in Java like what is included in Traveler. But this would also be an external program, which needs to be deployed.
For Ubuntu as the base image it would work with curl or better lftp out of the box.
Maybe someone has an additional idea.

Image:Mission impossible: sftp client with user/password support on Redhat UBI

Image:Mission impossible: sftp client with user/password support on Redhat UBI

Comments [0]

Proxmox Container

First Look - Proxmox 9.1.2 OCI Container support

Daniel Nashed – 10 December 2025 20:22:22

This is brand new and before someone asks: This is not ready for prime time.
You are better of using a Docker host on with a VM or LXC container.

The OCI runtime is not ready for prime time. Specially not for a container like Domino.

First of all pulling from a registry does not support authentication from what it looks like.
So you can't pull from a public registry that needs authentication.

But it would work with a private registry.
I first pulled simple images like busybox to test -- which worked well.
For Domino I exported the image from a Docker host in the following way and copied it into /var/lib/vz/template/cache/ docker save hclcom/domino:latest -o domino-docker.tar skopeo copy docker-archive:domino-docker.tar oci-archive:domino-oci.tar

Having it copied there let me select it via the LXC menu.
As a volume I selected a mount to /local. But the permissions have not been set right (the data directory was owned by root).
Also when you jump into the container you are always root first.

This isn't what we know from other container run-times and there have been a couple of other smaller issues I had to work-around.
The HCL out of the box image with Redhat UBI did not get an IP address via DHCP. But my Ubuntu image got an IP.

I was able to setup Domino and it is just running fine.

Still this isn't anything you want to use today for Domino.
It is a "don't try this at home" configuration until they improve it.

I just looked into it because someone asked and I wanted to give it a quick try.

On Proxmox the better way would be a LXC container or a VM with a Docker host.

If you want to take a look, use a simple container like busybox first.

Image:First Look - Proxmox 9.1.2 OCI Container support

Image:First Look - Proxmox 9.1.2 OCI Container support

Comments [0]

Domino Linux Container

Domino Diagnostics Collection on Linux

Daniel Nashed – 9 December 2025 22:52:08

Every customer situation can bring ideas for new functionality in the Domino Linux Start Script and container image.
I added a Domino Diagnostic menu earlier.

By default it did collect IBM_TECHNICAL_SUPPORT data from the last 24 hours.

This container almost all information. But could lead to collecting duplicate data collection.
Now the start script diagnostics will remember last diagnostics information which has been already sent.

The diagnostic compressed tar file will contain either data for the last 24 hours or since the last collection if this is a shorter time range.

In addition the tar file contains the translog control file. it will be copied with a time stamp in the name.
The idea is to have a very straightforward way to collect the right data for any kind of troubleshooting.
The diagnostic menu also got beefed up.

Image:Domino Diagnostics Collection on Linux

Image:Domino Diagnostics Collection on Linux

Comments [0]

Domino Container Domino Start Script

New Domino Container update options

Daniel Nashed – 7 December 2025 15:27:57

The Domino Container script is already a quite convenient way to install Domino and to maintain it.
In combination with the start script projects, software can be downloaded from MHS automatically and dominoctl helps to download web-kits.

The container project can be also used to install Domino natively -- with the same build script which is used to build the container image.
The build script is basically "just" executed on the Linux machine instead of in a build container.
There is also a convenience script to install Domino natively with one command.

But that command did not clone the GitHub repositories and just extracted them after download.
Nothing is every perfect. The container project and the start script projects are constantly improving.
Here are two new changes currently under test in the develop branch:

1. Enhanced Domino On Touch Installer to work for updates and clone GitHub repositories
The Domino on Linux installation script -> https://nashcom.github.io/domino-startscript/install_domino/ now clones the GitHub repositories and can be used also to update Domino servers after installation with the convenience install script.

2. Introduced a new update helper script and "update" commands
- Updates GitHub repositories locally by pulling updated
- Checks if any of the installed scripts need an update (dominoctl, domino start script, domdownload).

The script is linked to the main scripts: build.sh, dominoctl, domino start script. with the "-update" option to update the repositories and installs script updates.
A new "update" command ("updatescript for dominoctl") only updates and exits.

This new option makes it easier to update and not need to remember to pull repositories and update scripts manually.

Comments [0]

Domino Traveler Ubuntu

Issues with Traveler timezone on Ubuntu

Daniel Nashed – 5 December 2025 15:22:55

This is really a weird issue. It is not 100% why this happens on Ubuntu but not on Redhat Linux.
The settings are the same. But it looks like the two platforms handle timezones a bit differently in the back-end.

I ran into it in Docker during automation testing and got a ping from a customer today.

Background:

Traveler uses Java code but is a Domino application. So there are multiple components playing together.
It is essential that both components: 1. The Domino core server code and 2. The Traveler Java code do see the exact same timezone information.
Therefore the Traveler team added a check to let the server only start if this all matches.

Error reported by Traveler at start-up time which lets Traveler fail to start.Traveler: SEVERE *system Timezone discrepency. Domino reports 'W. Europe' which supports daylight savings and Java reports 'Koordinierte Weltzeit' (Etc/UTC) which does not support daylight savings. Traveler: SEVERE *system This discrepency may result in calendar events being shifted on devices synchronizing with this server. Please alter these values to be equivalent and make sure all operating system and Domino server fixes related to daylight savings time have been installed. Traveler: SEVERE *system Traveler server could not be started. The exception was ComponentNotStartedException. Exception Thrown: com.lotus.sync.util.ComponentNotStartedException: Timezone discrepency.

Linux background & Solution:

There are two timezone settings. One is controlled via timedatectl on Linux native.
The other part is handled by the TZ variable.

In our case on a native Domino server on Ubuntu the TZ variable was not set.
Adding the variable isn't that simple on a Domino server started as a systemd service, because the normal export TZ will not work depending where you add it.

One way would be to add it into the systemd service configuration.
But I added another way today in the start script.

In the same way you could specify DOMINO_LANG which turns into LANG when the server is started, you can now specify DOMINO_TZ which is exported as TZ before the server is started by the start script.
For a container image you would set the TZ variable in your container definition.

In my case the variable looks like this for Germany:TZ=Europe/Berlin

How to check if is is set

The Domino start script writes a notes.log, which dumps all configuration including the environment used to start the server (along with security limits and other details).
This helps to find out what is going on.

---

For completeness here is the output from timedatectl (which only works on a Linux native server, not on a container). timedatectl Local time: Fri 2025-12-05 16:17:55 CET Universal time: Fri 2025-12-05 15:17:55 UTC RTC time: Fri 2025-12-05 15:17:55 Time zone: Europe/Berlin (CET, +0100) System clock synchronized: yes NTP service: active RTC in local TZ: no

A more general check ls -l /etc/localtime lrwxrwxrwx 1 root root 33 Dec 5 12:53 /etc/localtime -> /usr/share/zoneinfo/Europe/Berlin

The timezone should point to the right zoneinfo. But that alone isn't sufficient on Ubuntu. You need the TZ variable to be set.

Another important detail:

You should always use a NTP server via chrony. This is specially important on VMs which are not always on.

Comments [1]

VMware Docker Podman Rancher

Using Virtual Machines and Linux container for software testing

Daniel Nashed – 28 November 2025 20:27:08

Now that Notes and Domino 14.5.1EA1 shipped this week some admins & developers are looking for options to test out the new version without impacting their production environment.

This is more a challenge on client side then on server side.

If you have a notebook or Macbook where you can install software, there are a couple of good (and free options).
In a corporate environment often the only option is to use a remote VM.

But there are some good client choices if one of those tools can be installed in your environment.

Windows

For a quick test of software on Windows the Windows Sandbox is a very simple to use solution.
It can be configured with volume mounts and share the already running resources of your host Windows.
There can be only one Sandbox at a time and when you should down the Sandbox all data not hosted on a mount is gone.
But often for testing this is exactly what you want.

https://learn.microsoft.com/en-us/windows/security/application-security/application-isolation/windows-sandbox/

VMware Workstation Pro 25H2

VMware workstation is back in business and has a new version.
It is free for private and commercial use. You can install multiple VMs if you have sufficient resources and have the full flexibility of snapshots.

https://www.vmware.com/products/desktop-hypervisor/workstation-and-fusion

MacOS

On the Mac you have multiple options

VMware Fusion is one of them.
But there is also UTM

UTM

https://mac.getutm.app/

Both can be used to install either a MacOS or Windows.

For Windows VMs on older Intel Macs you would use standard Intel/AMD64 Windows.
For Apple Silicon you should always pick the Windows ARM edition -- which works well for Notes and Domino.

Avoid running Domino on Linux on Apple Silicon

Running Linux on the Mac sadly only works well with Intel hardware.
There are still issues with the JVM shipped with Domino.
This is true for native Linux and any kind of container environment like Docker and Rancher Desktop.

Domino servers on Linux

For Linux there are a couple of desktop options.
You can run it in the VM infrastructure of your choice like VMware Workstation Pro.

But there are more options to choose from.

Windows Subsystem for Linux (WSL).

One great platform is WSL.
It is part of Windows and well integrated with Windows.

https://learn.microsoft.com/en-us/windows/wsl/install

You can access Windows disks using a standard enabled mount (e.g. /mnt/c) or the other way round access Linux volumes via Windows explorer.

Beside running a native Domino server on WSL you can also install a Docker host on WSL to run a Domino container.

Running Domino in Containers on Linux

This is one of the easiest and most flexible options.

The Domino container project allows you to build a standard or customized containers -- also with version 14.5.1 EA1.

You can run container images on

WSL based Linux host with Docker or Podman
Docker Desktop
Rachner Desktop

https://www.docker.com/products/docker-desktop/
https://rancherdesktop.io/

Docker for Desktop is free for smaller companies, but commercial for larger companies.
But WSL and Rancher Desktop are great options to use as well.

Unless you are using Docker Desktop or Rancher Desktop for other work, it would make probably most sense to look into WSL.

Comments [0]

Domino Container

FIX Domino Container Image: tzdata was missing in UBI 10 minimal

Daniel Nashed – 28 November 2025 12:14:22

The container image uses Redhat UBI 10 minimal as the base image.
In version 9 it contained tzdata by default.

The newer version does not contain it and the build script did not install it.
I have added logic to try to always install tzdata to make sure this never happens again for any distro change.

But I also decided to go back to have the normal not the minimal image as the default.
Admins can still switch back to the minimal using -from=ubi-minimal

The standard UBI image is the safer bet and the resulting image isn't that much bigger.

The develop and the main branch are both updated.

This only affected the recent builds where UBI 10 minimal was used.
The issue occurred was that the timezone was not switched inside the container even specifying your timezone at build time.

See details here --> https://opensource.hcltechsw.com/domino-container/howto_tz_locale/

If you are on UTC there isn't any issue at all. But for everyone else this is an important fix.

-- Daniel

Comments [0]

CertMgr

Leveraging the potential of Domino CertMgr + What’s new in 14.5.1

Daniel Nashed – 27 November 2025 11:07:31

Domino CertMgr introduced in Version 12.0 is one of the modern corner stones of Domino security.

This includes the TLS Cache and certstore.nsf which is used for many other security features today.

It is built on open standards and can be customized and leveraged outside Domino.
Certificates are stored in PEM format in text fields.
When generating exportable keys, they are stored in plain text PEM format encrypted with the modern (PKCS#8 + PBKDF2 + AES-256-CBC) encryption.

CertMgr supports multiple flows:

Manual import
ACME protocol with HTTP-01 and DNS-01 challenges
Import/Export
MicroCA

See details her --> https://opensource.hcltechsw.com/domino-cert-manager/certificate_flows/

Those flows can be mixed and matched and can be also extended for your own needs.
Specially the manual flow is designed to allow to export CSRs to for example send it to an external CA.
And import certificates once issued by an external CA.

The past operation can be automated by just writing the certificate into the right field and setting the status of the TLS Credentials document to the right state.

Details of the TLS Credentials document are documented here --> https://opensource.hcltechsw.com/domino-cert-manager/tls_credentials_anatomy/

Once created exportable TLS Credentials can be used outside Domino and can be even automatically updated querying certificates remotely from CertMgr.
See details here --> https://opensource.hcltechsw.com/domino-cert-manager/deploying_outside_dominio/

For exporting and importing TLS Credentials there is a script lib with a documented LS2SCAP call-out to use it for your own export and import applications.

Auto filter complete, auto sort and auto complete functionality

All mentioned flows support the same "automatical" certificate handling.
In most applications you have to specify the right order, specify exactly the certificates needed and add the root and intermediate certificates.

With Domino CertMgr you don't need to care about any of this manually.

The certificate chain provided is automatically sorted
The sorting happens from checking the public key to match it against the leaf certificate.
From the leaf certificate the chain is built backwards up to the root certificate.
Any certificates which don't match the chain or are duplicated are automatically filtered out.

If the chain is not complete CertMgr looks up trusted roots stored in certstore.nsf trust store.
This works for root certificates and also for intermediate certificates.
The resulting chain should be always complete and in the right order.

This functionality is available since version 12.0 and available for all flows without separate configuration.

Using TLS Credentials outside of Domino

When using Domino managed TLS Credentials outside Domino it becomes essential to monitor expiration.
Certificates in certstore.nsf are monitored by CertMgr automatically. Certificates are updated where the flow allows it and statistics are updated with number of certificates in green, yellow, red state.

For external used TLS Credentials there is a separate health check by URL, which daily checks all configured URLs, updates statistics and optionally sends one summary mail with warning and errors:

https://help.hcl-software.com/domino/14.5.0/admin/secu_certificate_url_health_check.html

All the functionality described is intended to simplify certificate management and make Domino a first class citizen in certificate management.

New functionality in 14.5.1 EA1
There isn't much to add, but in 14.5.1 EA1 there are some smaller enhancement based on customer feedback.

e-mail attribute in manual and ACME flow

Some customers require an e-mail attribute in the common name for their internal certificate flows.
The field was always present since 12.0 but has not been exposed in the UI,
Now with 14.5.1 it is an official feature of CertMgr for manual flows and also the ACME protocol (even most public ACME servers ignore it, but internal customer flows might need it).

Custom Expiration in MicroCA flow

The Domino MicroCA started as a very simple private CA for the AppDevPack.
But meanwhile it is used for JConsole Certs and can be leveraged for Domino IQ and other flows.

It is very useful for internal certificates for example behind a reverse proxy.
Because certificates are automatically renewed, distributed by replication via certstore.nsf and instantly updated by the TLS cache, they are a very good fit for those type of scenarios.
But the MicroCA does not provide any CRL end-point or OCSP option.

To comply with modern requirements the MicroCA now supports custom certificate expiration. So you could set it even as low as 3 days today.
The MicroCA flow would auto renew it as soon the renewal interval has been reached.

There is a new field on the TLS Credentials form Validity period shown in MicroCA mode.
The internal field was already present since 12.0. But the field has now exposed in the UI and is officially supported.

Subject Key Identifier (SKI) and Authorized Key Identifier for MicroCA

Those have been two missing extensions expected from modern CAs.
They have been added in 14.5.1 as well to comply with modern requirements.

New certificates would get those extensions automatically beginning with 14.5.1 EA1.
A new MicroCA would get a SKI when created with 14.5.1 too.
If you have special requirements and integration ideas let me know.

-- Daniel

Comments [0]